Now that we've extracted the body text of an email from a single email, we need to do this for all our 1
emails and for that we need to create a function. The kind of function that we're going to create 2
in this lesson is a special type of function in Python called a generator function. 3
In other words we will create a function that reads all the files in a folder. 4
Now the functions that we've encountered so far run once and they return a value. 5
If you recall a standard Python function just has that "return" keyword and then it spits out a value 6
following whatever comes after that keyword. 7
And that's it. 8
And what this means is that a function needs to return all the results at once. 9
It needs to return all the results at the same time. 10
Now if we wrote a function that read all the emails and extracted the body text from all the 5000 e-mails 11
all at once, then we would have to return 5000 email bodies all at the same time as well. 12
Now if this sounds like a lot of work, then you're absolutely right. 13
And we don't have to write our Python code to do it this way. 14
There is an alternative and this is where generator functions come into play. In our Python notebook, 15
let's add a Markdown cell that reads "Generator Functions" and in the cell below we're going to go over 16
this advanced functional pattern that you're going to encounter every time you want to spit out a series 17
of values. 18
We're gonna be combining two very powerful programming tools. 19
The first one is loops 20
and the second one is this generator function. 21
So before we parse 5000 e-mails, let's go through a practice generator function. Starts out the same way 22
as every other function, with a definition - "def" keyword, then I'll give it a name "generate_ 23
squares" and then I'll give it maybe a capital N as a single parameter and then inside this function 24
I'll write a loop: "for my_number in range(N):", 25
this is gonna be up to N, "yield my_number**2". 26
This here is my generator function. 27
It will take in a single value, N, and it will run the loop N times. 28
Now, one difference that you'll notice is that we don't have a return keyword. 29
Instead we've got this other keyword here. 30
Yield. 31
Let's call this generator function to see how it behaves, then we'll talk a little bit more about the 32
syntax. 33
Having pressed Shift+Enter on the cell, you might think that all we have to do is call the function by 34
using its name like so "generate_squares(3)", say, and press Shift+Enter. 35
But in this case the output looks a bit unexpected. Instead of squaring say the number three, 36
what we get is a generator object. 37
So how do we call this function in a more useful way? 38
One thing we can do is wrap this whole thing in a loop and then you'll also see how this generator function 39
actually works. 40
So if I say "for i in generate_squares(3):", "print(i)", and then the comma, and then say at the end put 41
a little arrow in between the results and let's hit Shift+Enter now. 42
So this is interesting, right. 43
We get 0, 1, 4 and then each number is separated by little arrow here. 44
What's going on? Our loop will run three times because N is equal to three. 45
But what we're doing here is we're feeding in the values into our generator function one at a time - the 46
first value that we feed in is the value 0 and 0 squared is equal to zero. 47
Then we feed in the value 1, 1 squared is equal to, well, 1, then we feed and the value 2, 2 squared is 48
equal to 4. 49
And the amazing thing here is that this function using the yield keyword remembers where it left 50
off. 51
So let's change our argument here to the number 5 and see how this goes. 52
Now our sequence looks like this: 0, 1, 4, 9, 16. In contrast to the return keyword for a normal function where 53
the function basically exits with a value 54
and we're done for good, with the yield keyword 55
it's sort of exiting the function but it remembers the state where we had exited from. 56
So in this case we're iterating through our loop and it remembers the previous value that it was at 57
and we're starting from the point where we had yielded from. 58
But why is this interesting? 59
Why does this matter? At first glance it looks like we could achieve the very, very same thing with a 60
normal function that uses the return keyword instead of having these loops and iterating through a generator 61
function. 62
Why would we do this? 63
Well, here's the thing, with a generator function we don't have to do all the upfront work. 64
So in our case we've got 5000 e-mails that we have to pass. With a large dataset like that or an incredibly 65
long list, 66
it takes an incredible amount of computation to even produce a single value let alone thousands of them 67
at the same time. 68
So what we're going to do now is we're gonna apply this generator function to loop over and iterate 69
over all the files in our directory that holds onto the spam emails and then we're basically going to 70
parse one email at a time. 71
That's how we're going to use this generator function. 72
Let me add another Markdown cell here that reads "Email body extraction" and what we'll do here is we'll 73
define a generator function that walks over all the file names in a particular folder. 74
This is a function from the operating system. 75
Here's how we're going to use it. 76
So we'll wrap the whole thing in a function. 77
Yeah. 78
"def email_body_generator()", 79
and this is going to take a single parameter, namely the relative path to one of our folders, 80
the spam folder or the folder with the legitimate emails. 81
Now what we'll do is we'll write a loop. 82
We're going to say "for root, dirnames, filenames in walk(path):". 83
This walk function is where our operating system comes in. The walk function generates the file names 84
in a directory by walking the tree from the top to the bottom and it yields, 85
that's right, doesn't return, it yields a tuple, so three things consisting of the directory path which 86
is this first one here, the directory names which is the second one here and the file names, which is 87
this third one here. The directory path is obviously the path to our spam folder in this case. The directory 88
names are the sub directories which we're actually not going to use and the file names is the bit that 89
we're actually interested in. This is gonna be a list of names of all the files in our directory. 90
In other words, if we point this function to "easy_ham_1", then we're gonna get all 91
these file names right here. We're gonna get all the file names in this "easy_ham" directory. 92
This is what we're after. Now, the walk function is not inbuilt. It belongs to the os library. So let's 93
import it at the very top of our notebook. Scrolling up, we're going to say "from os import walk" and while we're 94
up here, we're also going to import something else that we're gonna be using in this function, namely 95
the join method. 96
So "from os.path import join". 97
Now let me hit Shift+Enter and scroll back down. Let me add a semicolon and let's write the inner part 98
of this loop. The inner part of this loop is going to make use of all the file names that we're retrieving 99
using the walk function. 100
So what we want to do with a single file is actually very, very similar to this bit of code that we've 101
written earlier, but since this function is going to return all the files to us, we're gonna have to tackle 102
each file one by one. 103
But let me copy this code nonetheless and then down here, we're going to add another loop, namely we'll say "for 104
file_name in filenames:" and then let's paste in this code. I'm going to select this bit 105
here and just hit Tab on my keyboard to indent it and make sure it's in the body off my inner loop and 106
then I'm going to have to make another change. We're not gonna be targeting our example file. We need 107
to be targeting a particular file in this list of file names. How do we get that? Well we'll say the file 108
path of a particular file 109
is gonna be equal to joining the route, which we're getting here from our outer loop, to a particular 110
file name that we're iterating over in our inner loop. So we'll say "combine the path for the root directory 111
with a file name that we're iterating over in our loop". And then in our open function, we can replace 112
example_file with filepath. 113
Everything here will remain the same. 114
The only thing that's gonna change is that we're not gonna be printing out the email body. 115
We want this function to spit out two pieces of information - 116
one is the file name and the other one is the email body. And this is where we're gonna use that yield 117
keyword once again. 118
So we'll say "yield file_name, email_body". 119
Now I know this bit of code looks very, very involved, but we've broken it down quite a bit in the previous 120
lessons already. 121
So for example, we know that this bit of code extracts an email body from a particular file and we know 122
that using the yield keyword, this function here will give us a result every time it loops over a particular 123
file in our directory. 124
So I'll spit out this file name, then I'll spit out this file name and this email body, then I'll spit out 125
this file name and this email body and so on. The only thing that's really new is this walk function 126
from the os library which spits out a tuple, which we're using in our loop and we're nesting a inner 127
loop inside this one here to go over all the files one by one. 128
Now that's half of the work done. 129
If we look back up here, we've essentially done this bit. We now need to write the second piece of code 130
that actually calls our generator function. 131
We need to write a loop that repeatedly calls our generator function. 132
Let's put the second piece of code inside a function as well. 133
So I'm gonna go down here and I'm going to call this function "dataframe from directory", 134
so "df_from_directory" and it's going to take two inputs, it's gonna take a 135
path and a classification; and by classification I just mean whether this email folder is going to contain 136
spam emails or legitimate emails. To create our data frame we will start out with two empty lists. 137
So I'll say "rows = []" and "row_names" is also equal to a 138
pair of empty square brackets. Our generator function is going to be called Inside a loop. 139
So we'll say "for file_name, email_body" which is what our generator function 140
is returning; "in email_body_generator" and then our generator function here needs 141
one input, namely a path, and by the way if you haven't pressed Shift+Enter on this it's a good idea to 142
do so now and once you've done that all we need to do is supply a path to our generator function as 143
an argument and I'm just going to feed through the path that is being passed into this data frame from 144
directory function to our generator function right here. Inside the loop, 145
we're gonna append our email bodies to our rows list, so I'll say "rows. 146
append({'MESSAGE': email_body, 147
'CATEGORY': 148
classification})". 149
So what I've done here is I've created a Python dictionary using the values that our generator function 150
spits out. Each time this loop runs it's gonna give us a file name and an email body and we're storing 151
this in a list where we're appending the email body one by one as it goes over the files. 152
Next we'll do something very similar for the row names. So "row_names.append( 153
file_name)". 154
Now, this dataframe from directory function here is gonna be a regular function. 155
It's not going to yield anything. 156
It's going to return a dataframe, so "pd" for pandas, ".DataFrame(rows, index = 157
row_names)" and that's it. 158
Except that we need to "import pandas as pd" at the top of our notebook of course. 159
So let's do that now "import pandas as pd", Shift+Enter, scroll down and hit Shift+Enter on this as well. 160
Now we've written quite a bit of code and we haven't tested it at all. 161
So we're not even sure if all of this will work or if we've made an error. 162
Let's try and call this df_from_directory function and see if it works. 163
But before we do that let's add all our paths to the top of our notebook 164
under this constants heading. The paths that I'm interested in are the paths to easy_ham_1, 165
the path to "easy_ham_2", "spam_1" and "spam_2". 166
So let's create four constants with these paths. The kind of path that we're gonna be working with. 167
is gonna be a relative path. The Bayes Classifier notebook is located under MLProjects, so our path 168
will have to go into SpamData, 169
01_Processing, spam_assassin_corpus, 170
and then we'll have these folder names afterwards. 171
So here we go. 172
"SPAM_1_PATH" is going to be equal to this first bit here, which I'm just gonna 173
copy and paste, 174
then we said it was "spam_assassin_corpus", 175
and then we said it was gonna be "spam_1". 176
This is the relative path from our Bayes Classifier notebook to our Spam_1 folder. 177
Now remember, everything is case sensitive and you've got to use forward slashes between the folder names 178
to avoid getting any errors. 179
Let's tackle the other four relative paths now. 180
So I'm just going to copy this, paste it three more times and rename our constants here as well. 181
So "SPAM_2_PATH"; this one I'll call "EASY 182
NONSPAM_1_PATH", and this one I'll call "EASY_NONSPAM_2_PATH". The folder that these ones 183
are going to point to 184
is gonna be "easy_ham_2", "easy_ham_1", "spam_2" and "spam_1" of course. 185
And that's it. 186
If we hit Shift+Enter on this and just make sure we haven't made any typos, we're good to go. Let's call 187
our df_from_directory and create a dataframe of spam emails. 188
So I'm going to call this dataframe "spam_emails" and I'm going to set it equal to "df_from_directory". 189
Note I don't have to type all this out, I can just hit Tab on my keyboard, "(SPAM_1_PATH,)" 190
and then the category for spam is going gonna be the number 1. Now before I continue going, 191
let's look at the head of this dataframe to check out the first few rows. 192
So "spam_emails.head()" and Shift+Enter. 193
Voila! Here we can see the file names of the first five rows. 194
We've got a category. 195
Category is going to be 1 for spam and 0 for non spam. 196
And then we've got the messages, 197
in other words the bodies of all the emails as a column in our dataframe as well. 198
Let's take a look at the shape of this dataframe to see if we've got all our emails, "spam_ 199
emails.shape", Shift+Enter gives us 501 and 2. Two for the number of columns, the 200
category and our email bodies and 501 for the number of rows, 201
in other words the number of messages in this folder.